feat(cluster): automatic failover via epoch-based voting - #189
Merged
Conversation
adds two new NodeUpdate variants — VoteRequest and VoteGranted — that are piggybacked on Ping/Ack messages to coordinate automatic failover elections. also adds the matching GossipEvent variants that the server layer consumes, and queue_vote_request / queue_vote_granted helpers on GossipEngine. the new election.rs module tracks election state: epoch, votes received, and quorum calculation. a majority (n/2 + 1) of alive primaries must vote for a candidate before it can promote.
when gossip confirms a primary as dead and this node is one of its replicas, we start an election: - wait 500ms (stagger delay lets more up-to-date replicas win) - broadcast VoteRequest via gossip piggybacking - primaries that haven't voted in this epoch respond with VoteGranted - first replica to reach majority quorum promotes itself via FAILOVER FORCE VoteRequest events are handled by primaries (one vote per epoch, enforced by last_voted_epoch atomic). VoteGranted events are handled by the candidate; quorum triggers cluster_failover(force=true). post-lock actions (StartElection, HandleVoteRequest, HandleVoteGranted) are dispatched from the gossip event consumer after releasing the state write-lock to avoid deadlocks.
kacy
added a commit
that referenced
this pull request
Feb 19, 2026
* feat(cluster): add VoteRequest/VoteGranted gossip updates for elections adds two new NodeUpdate variants — VoteRequest and VoteGranted — that are piggybacked on Ping/Ack messages to coordinate automatic failover elections. also adds the matching GossipEvent variants that the server layer consumes, and queue_vote_request / queue_vote_granted helpers on GossipEngine. the new election.rs module tracks election state: epoch, votes received, and quorum calculation. a majority (n/2 + 1) of alive primaries must vote for a candidate before it can promote. * feat(server): automatic failover via epoch-based voting when gossip confirms a primary as dead and this node is one of its replicas, we start an election: - wait 500ms (stagger delay lets more up-to-date replicas win) - broadcast VoteRequest via gossip piggybacking - primaries that haven't voted in this epoch respond with VoteGranted - first replica to reach majority quorum promotes itself via FAILOVER FORCE VoteRequest events are handled by primaries (one vote per epoch, enforced by last_voted_epoch atomic). VoteGranted events are handled by the candidate; quorum triggers cluster_failover(force=true). post-lock actions (StartElection, HandleVoteRequest, HandleVoteGranted) are dispatched from the gossip event consumer after releasing the state write-lock to avoid deadlocks.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
implements automatic failover for primary nodes that gossip confirms as dead.
replicas of the failed node start an epoch-based election, collect votes from
a majority of the remaining primaries, and promote themselves without manual
intervention.
crates/ember-cluster/src/election.rs(new) — election state machinetracking epoch, votes received, and quorum (
n/2 + 1)crates/ember-cluster/src/message.rs— two newNodeUpdatevariants:VoteRequest { candidate, epoch, offset }(tag 7) andVoteGranted { from, candidate, epoch }(tag 8), both serialised in theexisting binary gossip wire format
crates/ember-cluster/src/gossip.rs— matchingGossipEventvariants;apply_updates()emits them;queue_vote_request/queue_vote_grantedhelpers on
GossipEnginecrates/ember-server/src/cluster.rs— three new methods onClusterCoordinator:start_election(failed_primary)— stagger delay + broadcast vote requesthandle_vote_request(candidate, epoch)— primaries grant one vote per epochhandle_vote_granted(from, candidate, epoch)— candidates track quorum,trigger
cluster_failover(force=true)when reachedthe gossip event consumer detects
MemberFailedfor our primary and spawns anelection task. vote events are handled via a
PostActionenum so the statewrite-lock is released before calling gossip or async failover methods.
What was tested
ember-clusterunit tests pass (gossip, message roundtrips,election state machine, topology)
ember-serverunit tests pass, including four new election tests:primary_grants_vote_once_per_epochreplica_does_not_grant_votevote_granted_reaches_quorum_and_promotesvote_granted_wrong_candidate_ignoredcargo clippy --workspace -- -D warningscleanDesign considerations
stagger delay is a fixed 500 ms today. in production, the delay should be
proportional to
(max_offset - my_offset)so the most up-to-date replica wins.this requires either a gossip-advertised max offset or a coordination round;
left as a follow-up since the correctness of the voting algorithm does not
depend on it.
vote scope is intentionally limited to primaries. replicas that are not
candidates never vote, preventing split-brain from out-of-date nodes.
epoch enforcement (
last_voted_epochatomic) ensures each primary votesat most once per config epoch, regardless of how many VoteRequest messages
it receives from competing candidates.